Correlation Effects in Quantum Spin-Hall Insulators: A Quantum Monte Carlo Study 
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We consider the Kane-Mele model supplemented by a Hubbard U term. The phase diagram is 
mapped out using projective auxiliary field quantum Monte Carlo simulations. The quantum spin 
liquid of the Hubbard model is robust against weak spin-orbit interaction, and is not adiabatically 
connected to the spin-Hall insulating state. Beyond a critical value of U > U c both states are 
unstable toward magnetic ordering. In the quantum spin-Hall state we study the spin, charge and 
single-particle dynamics of the helical Luttinger liquid by retaining the Hubbard interaction only 
on a ribbon edge. The Hubbard interaction greatly suppresses charge currents along the edge and 
promotes edge magnetism, but leaves the single-particle signatures of the helical liquid intact. 

PACS numbers: 03.65.Vf,71.10.Pm,71.27.+a,71.30.+h 



The Z2 topological band insulator (TBI) [1 arises from 
spin-orbit (SO) coupling and is invariant under time re- 
versal symmetry. The bulk is insulating and the edge 
states, coined helical Luttinger liquids, show gapless spin 
and charge excitations. An explicit realization is given 
by the Kane-Mele (KM) Hamiltonian [2] which reduces 
to two separate Haldane models [3], with opposite signs 
of the Hall conductivity in the two spin sectors. Time 
reversal symmetry protects the edge states against po- 
tential scattering and weak electron-electron interactions 
[HE], and allows for experimental realizations [6j[7]. Pre- 
vious work on correlation effects has essentially followed 
two routes: interaction driven topological insulators [8]- 
[T2] . or (as here) the interplay of spin-orbit coupling and 
Coulomb repulsion [T3HT7] . We present the first quan- 
tum Monte Carlo (QMC) results which document (i) a 
quantum phase transition between the quantum spin liq- 
uid (QSL) phase of [18] and the TBI, (ii) the stability of 
the TBI against magnetic ordering, and (Hi) the role of 
fluctuations in the helical edge states of the TBI. 

Our starting point is the KM-Hubbard model on the 
honeycomb lattice with Hamiltonian H = Hkm + Hu , 
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The spinor c\ = (cj"^,Cj^) creates an electron in a 
Wannier state at site i, means summation over 

the three nearest neighbors j = i + 6 n with S n G 
{±<Si, ±£2, ±£3}, see Fig. [lja), denotes sum- 

mation over next-nearest neighbors j = i + £ n + £ m , 
^ij = S n x S m /\S n x S m \ and a is the vector of Pauli 
matrices. At the particle- hole symmetric point, this 
model can be investigated with a variety of QMC algo- 
rithms without encountering the infamous negative sign 
problem. We present two sets of simulations to extract 
bulk and boundary properties. 

Bulk phase diagram. — For bulk simulations we use the 
projective auxiliary field QMC approach. The ground 



state |^o) is filtered out of a trial wave function \^t) with 
(^t|^o) 7^ 0; a very good choice is the ground state of 
the KM model. For an arbitrary observable (^/o\0\^/o) = 
lim e ^oo(^T|e- ^ /2 Oe- 0// / 2 |^T)/(^T|e- ^|^T). The 
absence of the negative sign problem at half filling follows 
from the fact that after a discrete Hubbard-Stratonovich 
transformation of Hu and subsequent integration over 
the fermionic degrees of freedom, the fermionic deter- 
minants in the up and down spin sectors are linked via 
complex conjugation such that their product is positive. 
We employ an SU(2) invariant Hubbard-Stratonovich 
transformation and an imaginary time discretization of 
Art = 0.1. Projection parameters Qt = 40 prove suf- 
ficient for converged (within statistical errors) ground- 
state results. For details of the algorithm see [19]. 

The SO coupling reduces the SU (2) spin symmetry to 
a U(l) symmetry corresponding to spin rotations around 
the z-axis. The Hubbard interaction promotes trans- 
verse, x-y magnetic ordering [15] which can be tracked by 
computing the antiferromagnetic (AFM) structure factor 
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on L x L honeycomb lattices with periodic boundary 




FIG. 1: (Color online) (a) Periodic lattice structure of the 
KM-Hubbard model with nearest-neighbor hopping t, spin- 
orbit coupling A and Coulomb repulsion U. Arrows indicate 
the current direction associated with the spin-orbit term for 
one spin species and sublattice. (b) Effective model on a semi- 
infinite ribbon with periodic boundaries in the ai direction, 
and Coulomb repulsion U only at the edge sites. 
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FIG. 2: (Color online) Phase diagram of the KM-Hubbard 
model from QMC simulations. Bullets correspond to com- 
puted phase boundaries. The four phases are the semi metal 
(SM), the quantum spin liquid (QSL), the topological or 
quantum spin-Hall insulator (TBI), and the antiferromagnetic 
Mott state (AFM-MI). Inset: Ground-state spin-spin correla- 
tion function [Eq. pi] for an L x L honeycomb lattice with 
periodic boundary conditions and X/t = 0.25. Lines are fits 
to the form a + b/L + c/L 2 . Negative values indicate that 
the data decay quicker than 1/L 2 . Here and in subsequent 
figures, errorbars are omitted if smaller than the symbol size. 
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FIG. 3: (Color online) (a),(b) Single-particle gap along dif- 
ferent cuts in Fig. [2] (A c /£ ~ 0.03, U c /t ~ 4.9). (c) Raw data 
and size extrapolation (inset) at U/t = 4, X/t = 0.03. 



conditions. At X/t = 0.25, this quantity is plotted ver- 
sus lattice size for various values of U/t in the inset of 
Fig. [2] The onset of long-range order occurs in the region 
6 < U c /t < 6.25. Because of the underlying U(l) symme- 
try the quantum phase transition between the magneti- 
cally ordered and disordered phases is expected to be in 
the 3D-XY universality class. Figure [2] shows U c /t as 
a function of X/t and thus defines the magnetic phase 
diagram. 

Several aspects of Fig. [2] deserve comments: (i) With 
the important exception of the QSL phase, the quali- 
tative aspects of the magnetic phase diagram were ob- 
tained at the mean- field level [15 j. The magnetic insta- 
bility smoothly converges to the X/t = result [18 . For 
0.025 < X/t < 0.25 we observe no spin ordering along the 
z quantization axis up to U/t = 9. (ii) The U/t = line 
corresponds to the KM model and describes a TBI with 
a single-particle gap set by A. (Hi) The X/t = line has 
been investigated in detail in Ref. [18 . Up to U/t = 3.5 
the semimetallic (SM) phase remains stable and magnetic 
order sets in from U/t = 4.3 onwards. The intermediate 
phase shows both spin and single-particle gaps and cor- 
responds to a QSL. 

To further investigate the phase diagram and in partic- 
ular the evolution of the QSL upon switching on the SO 
coupling, we have computed the single particle gap, A sp , 
at the Dirac point K. This quantity is extracted by fit- 
ting the tail of the single-particle imaginary time Green 
function, G(K,r) = <*o|4,»<«,J*o> [see Fig.gc)], 



to the form Ze _rAsp , where Z corresponds to the quasi- 
particle residue. The extrapolated (in L) value of A sp is 
plotted in Figs. [3|a),(b) along different cuts of the phase 
diagram. Starting in the SM phase, at U/t = 2, A sp ex A 
as for the U/t = case and characteristic of the TBI 
state 0. In contrast, in the QSL phase at U/t = 4, A sp 
initially decreases with increasing A but grows again for 
\/t> 0.03. We interpret this cusp feature as a signature 
of a quantum phase transition between the QSL state 
and the TBI at X c /t ~ 0.03. The data support the van- 
ishing of the single-particle gap at X c /t [20]. Figure |3|b) 
shows that the magnetic transition as a function of U/t at 
fixed X/t = 0.1 is equally apparent; A sp smoothly evolves 
from its U/t = value and exhibits a cusp feature at 
U c /t ~ 4.9. Magnetic order breaks time reversal sym- 
metry and lifts the topological protection, so that A sp 
does not have to close at U c . This evolution of the gap 
can be qualitatively reproduced at the mean-field level. 
From the single-particle gap, we identify four distinct 
phases: (i) a TBI phase, where A sp evolves smoothly to 
its U/t = value, (ii) a magnetically ordered MI, (Hi) a 
SM line and (iv) a QSL phase. 

Edge states in the TBI phase. — Edge states are a hall- 
mark feature of TBIs. A detailed understanding of cor- 
relation effects in these one-dimensional liquids is crucial 
for theory and experiment. To study the helical Lut- 
tinger liquid formed at the edge of the Z2 TBI, we con- 
sider the ribbon topology of Fig.JTJb). For U > U c , time 
reversal symmetry is broken spontaneously and scatter- 
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FIG. 4: (Color online) Transverse spin correlations along the 
edge of the ribbon. Results are for L — 16, fit — 20 (dashed 
lines) and fit = 40 (solid lines). Lines are guides to the eye. 

ing between the left spin-down, and right spin-up movers 
of the helical liquid is allowed, thus opening a gap in 
the edge states and destroying the TBI state. As argued 
above, at U < U c the bulk is adiabatically linked to the 
U/t = line. Since furthermore the helical liquid is ex- 
ponentially localized on the boundary (as readily seen in 
the KM model), we retain the Hubbard interaction only 
on one zig-zag edge of the ribbon [cf. Fig. []Jb)]. With 
this ansatz, the bulk plays the role of a fermionic bath 
which can be integrated out at the expense of a Gaussian 
integral. This yields an effective one-dimensional action, 

S = - £ f dr f dr'cj. ff G^(r - r')c r , >a 

+ T Kt(r) - |] K,(r)-i] , (3) 

r Jo 

where r is an edge site index, and Go, a (r — r') is the 
free Green function of the KM model on the ribbon 
topology. We can solve the action ([3| exactly using the 
weak-coupling expansion continuous-time QMC method 
[21j[22], on arbitrarily wide LxL' ribbons (here V = 64). 
The validity of this effective model at U < U c has been 
verified by QMC calculations for the full model ([!]) on 
narrow ribbons. We take X/t = 0.25 in the following. 

At U/t = and half filling, the dispersion relation 
of the helical liquid satisfies e q ^ = — s q ^, and the edge 
states are unstable towards transverse ferromagnetic or- 
der. Figure [4] shows the development of substantial spin- 
spin correlations in the transverse direction with decreas- 
ing temperature and increasing U/t. This corresponds to 
the dominant correlation function. 

We calculate dynamic structure factors along the edge, 

O(q,oj) = |^e-^«Km|0( 9 )|n)| 2 5(£; ro -^-a;). (4) 

n,m 

For charge, N(q,u), O(q) = N(q) = ^E r e i(?r n r , 
and for spin, S a (q,u>), O(q) = S a (q) = ^Er e ^- 
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FIG. 5: (Color online) Dynamic spectral functions in the (a) 
charge sector, (b) spin-resolved one-particle sector, and (c,d) 
spin sector, measured along the edge. The parameters are 
U/t = 2, L = 24, L' = 64, X/t = 0.25 and fit = 40. Dotted 
lines show the velocities of the free helical liquid (U/t = 0). 

Single-particle dynamics are deduced from the single- 
particle Green function via the spectral functions 
A a (q,uj) = — 7r~ 1 ImG a (q^ cj), where by time reversal 
symmetry we have A^(q,ui) = A^(—q,ui). 

Figure |5] shows these dynamic quantities at U/t = 2. 
The dominant features of the single-particle spectral 
function, see Fig.[5|b), follow the noninteracting system: 
within the bulk band gap, gapless single-particle exci- 
tations emerge with a velocity tied to the z-component 
of the spin. For U/t = 0, the particle- hole spectra can 
be deduced from the single-particle dynamics by com- 
puting the bubble. Within this framework the dynamic 
charge structure factor as well as the z-component of the 
dynamic spin structure factor are identical. Both quan- 
tities conserve the z-component of spin, such that at low 
energies (i.e. below the bulk gap) only particle-hole ex- 
citations within the left, or right movers are allowed. 
This produces a linear mode around q = as observed 
in Figs. [5ja),(d). At higher energies, particle- hole exci- 
tations involving bulk states become apparent. Upon 
inspection of Figs. [5ja),(d) one sees that the support 
of both quantities is very similar. However, the spec- 
tral weight of the low-lying charge modes is greatly sup- 
pressed in comparison to the longitudinal spin mode. The 
transverse spin susceptibility involves a spin-flip process 
and hence excitations between the left and right disper- 
sion relations. This produces a continuum of excitations 
in the long- wavelength limit [cf. Fig. [5jc)]. 

At large U/t = 5 (Fig. [6| we observe a strong de- 
pletion of spectral weight in the low-lying charge modes 
[Fig. [6ja)], which leads to reduction of the Drude weight 
by 1 order of magnitude. In contrast, despite strong cor- 
relations, the single-particle spectrum [Fig.[6^b)] still ex- 
hibits the typical signature of the helical edge state. The 
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FIG. 6: (Color online) Same as in Fig. [5]but for U/t = 5. 

growth of the equal-time transverse ferromagnetic corre- 
lations as a function of U/t (Fig. |4| leads to a piling up 
of low- lying spectral weight in S x (q 1 uu) for q — >• 0. 

In the TBI phase, where the effective model of Eq. (|3| 
is valid, one can argue that the charge, longitudinal spin 
and Fermi velocities should be rather insensitive to the 
value of U/t since they are inherited from the bulk. This 
is confirmed by our numerical results. Correlation effects 
become manifest in a very strong variation of matrix ele- 
ments in the dynamic quantities of Eq. Q. In particular, 
the depletion of low-lying spectral weight in the charge 
sector suppresses charge transport along the edge. In 
contrast, spin fluctuations as well as the signatures of 
the helical liquid in the single-particle spectra persist. 

Summary. — We have derived the bulk phase diagram 
of the Kane-Mele Hubbard model from QMC simulations. 
We established the exact location of the previously pre- 
dicted magnetic transition at large U/t [15] , and that 
the nonmagnetic region is dominated by the TBI. The 
single-particle gap provides strong evidence for a quan- 
tum phase transition at finite SO coupling between the 
QSL and the TBI. Neither of these states can be charac- 
terized by a local order parameter, and a detailed under- 
standing of the transition represents a fascinating open 
issue. Applying QMC to an effective model of the heli- 
cal edge states, we have studied the impact of electronic 
correlations by calculating one and two-particle dynam- 
ics in the TBI phase. Correlation effects lead to an or- 
der of magnitude reduction of low-lying long wavelength 
charge fluctuations, and thereby charge transport, and 
promote transverse magnetic fluctuations. The single- 
particle spectrum retains its weak-coupling features. 
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Note added. — After completion of this work (arxiv: 
1011.5063), a QMC investigation [23] and two approx- 
imate studies [24, 25 of the same model came out. 
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